True Peer Review

In computer science, conferences and journals conduct peer review in order to decide what to publish. Many have pointed out the inherent weaknesses in peer review, including those of bias, quality, and accountability. Many have suggested and adopted refinements of peer review, for instance, double blind peer review with author rebuttals. In this essay, I argue that peer review as currently practiced conflates the sensible idea of getting comments on a paper with the irrevocably-flawed one that we either accept or reject the paper, which I term gatekeeping. If we look at the two separately, then it is clear that the ills associated with current peer review systems are not due to the practice of getting comments, but due to the practice of gatekeeping. True peer review constitutes my proposal for replacing existing peer review systems. It embraces the idea of open debate on the merits of a paper; however, it rejects unequivocally the exercise of gatekeeping. True peer review offers all the benefits of current peer review systems but has none of its weaknesses. True peer review will lead to a truly engaged community of researchers and therefore better science. I. THE DEBATE ON PEER REVIEW “You just have to resubmit and hope to get assigned the right set of reviewers,” advised an experienced mentor with whom I was discussing ways of improving a recently rejected paper of mine. In other words, keep trying until you get lucky. Whereas the advice was well-meaning, it betrayed a lack of trust in the peer review process. Others have echoed a similar sentiment. Naughton [6] in a recent well-publicized keynote mentions the large role a lucky assignment of reviewers plays in getting a paper accepted. Anderson [1] backs up this claim with statistical evidence from computer systems-related venues. Many have noted the problems with traditional peer review. Casati et al. [2] criticize the current publication model for entangling the separate concerns of dissemination, evaluation, and retrieval. In his keynote, Naughton noted the problems resulting from the combination of the pressure to publish, low acceptance rates, and poorly-trained reviewers, including that of undue negativity in reviews . More commonly noted are the problems of bias and accountability and that most speculative, potentially interesting research tends to get rejected in favor of incremental work [3]. A survey of researchers undertaken on behalf of the Publishing Research Consortium (PRC) contains extensive pointers to the ongoing debate on the efficacy of peer review [10]. Researchers in computer science recognize some of the limitations of peer review, and they are changing their systems to mitigate them. For example, the AAMAS (Autonomous Agents and Multiagent Systems) series of conferences implement double blind paper reviewing with author rebuttals. Further, senior program committee members and the program chairs monitor the quality of the reviews. Some conference series such as ICSE (Software Engineering) and RE (Requirements Engineering) have separate tracks for vision papers and new ideas and emerging results. The VLDB Foundation no longer publishes conference proceedings: all papers accepted to the foundation’s journal are presented at the next VLDB conference. Some have adopted more open systems of peer review in order to counter the problem of accountability, for example, the now defunct Electronic Transactions on Artificial Intelligence. One of Naughton’s proposal for improvement is not conducting peer review at all and accepting everything. Besides the engineering of peer review systems, researchers have also attempted to educate potential reviewers and writers on their respective tasks [9], [7]. Reviews forms at most conferences and journals are increasingly detailed, ostensibly to make sure that reviewers not overlook any major quality of the paper. While these are all well-intentioned efforts, they miss the point: we must get out of the accepting-rejecting business altogether and instead embrace the true spirit of scientific engagement, which I term true peer review. The rest of this essay is an elaboration of this point.